1. Field of the Invention
The present invention relates to an imaging device and an imaging method for forming subject images on an image sensor, which have respectively passed through different areas of a photographic lens in two or more directions, to acquire different viewpoint images. The present invention is more particularly relates to an imaging device and an imaging method for simultaneously imaging a stereoscopic image having parallax and a plane image.
2. Description of the Related Art
Conventionally, there has been a stereoscopic imaging device configured to form subject images, which have respectively passed through different areas of a photographic lens in two directions, on an image sensor to acquire different viewpoint images.
An optical system shown in FIG. 10 is configured to cause a mirror 4 to pupil-divide subject images that have passed through different horizontal areas of a main lens 1 and a relay lens 2, and to form the respective subject images on image sensors 7 and 8 through image formation lenses 5 and 6.
(A) to (C) portions of FIG. 11 are views showing the state of separation of images formed on the image sensor in the case of front focus, in-focus (best focus), and rear focus states. Note that in FIG. 11, the mirror 4 shown in FIG. 10 is omitted in order to compare differences in image separation depending on the focus states.
Among the pupil-divided images, the in-focus images are formed at the same position (are aligned) on the image sensor as shown in (B) portion of FIG. 11, whereas the front-focus and rear-focus images are formed at different positions (are separated) on the image sensor as shown in (A) and (C) portions of FIG. 11.
Therefore, by acquiring subject images which have been pupil-divided in the horizontal direction through the image sensors 7 and 8, it becomes possible to acquire a left viewpoint image and a right viewpoint image (3D image) having different viewpoints depending on subject distances.
According to Japanese Patent Application Laid-Open No. 2007-104248 (hereinafter referred to as Patent Literature 1), an electronic camera has an imaging unit, a light amount distribution detection unit, an image shift amount detection unit, and an image processing unit. The imaging unit is configured to photoelectrically convert a subject image obtained by the photographing optical system to generate photographed image data. The light amount distribution detection unit detects respective light amount distributions of luminous fluxes which have come from the subject and have passed through different optical paths. The image shift amount detection unit detects an image shift amount in a plurality of points within a photographing screen based on the light amount distributions. The image processing unit is configured to change, in the horizontal direction, positional relationship of the subject in the photographed image data based on the image shift amount in the plurality of points to generate stereogram image data. The image processing unit is also configured to change, in the horizontal direction, the positional relationship of the subject in the stereogram image data based on an input from an input unit to correct the stereogram image data. Since the stereogram image data can be corrected, it becomes possible to acquire a natural stereogram image, which causes less discomfort for users.
Japanese Patent Application Laid-Open No. 2009-124308 (hereinafter referred to as Patent Literature 2) discloses a method for generating a multi-viewpoint image, including: performing pattern matching between left-eye and right-eye stereo images photographed with two cameras on the left side and right side, using one of the stereo images as reference; specifying a matched image for each pixel; calculating an intermediate position image by interpolating an interpixel distance between images matched for each of the left and right pixels; and calculating a right outer image and a left outer image by extrapolating the interpixel distance between the images matched for each of the left and right pixels.
In Japanese Patent Application Laid-Open No. 2008-92007, paragraphs 0047 to 0048, and 0071 (hereinafter referred to as Patent Literature 3), a stereo matching processing unit (object detection means) detects one or more corresponding points (objects) corresponding to each other in search spaces set by the aforementioned space setting unit in two pieces of image data A and B which have been each processed by image processing units and stored in a memory section. A distance calculation unit (position calculation means) calculates three-dimensional coordinate values (position information) of the corresponding points detected by the above-stated stereo matching processing unit. Photographic lenses in photographing units are different in configuration. The photographic lens includes a zoom lens and an unshown zoom lens driving unit (driving means) that drives the zoom lens, while the photographic lens includes a fixed-focus lens with an angle of view equal to a wide angle end of the zoom lens of the photographic lens. Such configuration is employed to achieve cost reduction.
Japanese Patent Application Laid-Open No. 2003-7994, Japanese Patent Application Laid-Open No. 2001-12916, and Japanese Patent Application Laid-Open No. 2001-016611 (hereinafter referred to as Patent Literatures 4 to 6) are examples of a three-dimensional image generation technology with use of a single optical system. For example, Patent Literature 4 discloses a solid-state image sensor configured to include a large number of pixels arrayed on the same imaging surface and to photoelectrically convert a subject image formed on the imaging surface so as to generate an image signal. In the solid-state image sensor, the large number of pixels is divided into two groups, and pixels in the respective groups are made to have different light incident angles.
Japanese Patent Application Laid-Open No. 08-331607, Japanese Patent Application Laid-Open No. 2008-141666, Japanese Patent Application Laid-Open No. 2009-14445, Japanese Patent Application Laid-Open No. 2008-116309, and Japanese Patent Application Laid-Open No. 2000-102040 (hereinafter referred to as Patent Literatures 7 to 11) disclose examples of a method for searching corresponding points between different viewpoint images, a technology for acquiring depth information by stereo matching, and a three-dimensional image generation technology involving use of two-dimensional images and distance information (depth information).
In Japanese Patent Application Laid-Open No. 2008-271241 (hereinafter referred to as Patent Literature 12), the focus is placed only on a main subject and portions other than the main subject are intentionally defocused. Accordingly, a plurality of images are obtained by successively photographing images while discretely moving focal positions by a predetermined moving amount, and the thus obtained images are synthesized so that a blur-emphasized image can be obtained.
In Japanese Patent Application Laid-Open No. 2003-209727 (hereinafter referred to as Patent Literature 13), an image to be photographed is divided into a plurality of areas, and the areas, which are to be a foreground and a background of a main subject, are imaged with a focal position shifted from a focal position corresponding to a distance of the areas. Images of respective areas are individually extracted from the images imaged for each area, and these images are synthesized to form one image. This makes it possible to easily photograph an image with a desired blur amount.
Japanese Patent Application Laid-Open No. 2010-226500 (hereinafter referred to as Patent Literature 14) shows one example of a method for generating a parallax map.
In Japanese Patent Application Laid-Open No. 2003-209858 (hereinafter referred to as Patent Literature 15), parallax values in a parallax value distribution image are linearly or nonlinearly converted for parallax value adjustment, so that parallax value configuration is changed to depth value configuration of multi-viewpoint stereoscopic images, which is adapted to simultaneously fulfill both conditions for stereoscopic image designing, that is, “adopting a viewpoint which effectively uses a maximum amount of protrusion and a maximum amount of depression while limiting these amounts within a certain maximum parallax range” and “nulling parallax at a main subject position in an image as much as possible” and to effectively use the maximum parallax value range. By using a modified parallax value distribution image and viewpoint positions, i.e., by using a ratio r of each of a pair of inputted right and left stereo images to the parallax between these images corresponding to the viewpoint positions, a virtual viewpoint image of each view position is generated, and thereby a new viewpoint image all the pixels of which are effective is generated. This processing is repeated for the number of viewpoints as a multi-viewpoint image sequence. Three-dimensional strip images are synthesized from the multi-viewpoint image sequence. In this case, a three-dimensional image is synthesized so that pixels at the same coordinates in respective images in the multi-viewpoint image sequence are arranged as adjacent pixels in accordance with viewpoint placement of the images. In the synthesizing processing, the image of each viewpoint is cut into strips per line in a vertical direction, and the strips are combined for the number of each viewpoint in a reverse order of the viewpoint positions. Here, the images are synthesized in the reverse order of the arrangement of viewpoint positions because when images are observed through a lenticular board, left and right images are reversely observed in one lenticular pitch. If the lenticular board is overlapped with an image printed by this processing, a successful stereoscopic image can be observed.
A 3D image output apparatus according to Japanese Patent Application Laid-Open No. 2010-206774 (hereinafter referred to as Patent Literature 16) selects a parallax adjustment parameter in accordance with a screen size or viewing distance of a 3D display, converts the parallax amount at each characteristic point with use of a selected conversion table, generates a parallax image corresponding to the converted parallax amount, and outputs a plurality of parallax images including the generated parallax image to a display device. The viewing distance is automatically acquired by a distance measurement means provided on a 3D display or adjacent thereto or manually inputted. The 3D image output apparatus outputs parallax images to a stereoscopic display device which has various screen sizes, a printer which generates stereoscopic display prints, and the like.
In a pupil dividing-type stereoscopic imaging device configured to image parallax images by selectively acquiring image information that have passed through different pupil positions in image formation means, to generate a stereoscopic image as in Patent Literatures 4 to 6, parallax becomes 0 at the in-focus position, while at the out-of-focus position, parallax is generated corresponding to blurring (defocus amount).
Accordingly, in order to obtain a plane image without blurring in the stereoscopic imaging device, it is necessary to photograph an image with a narrowed diaphragm aperture and a deeper focus depth. However, if photographing is carried out with a narrowed diaphragm aperture, it is not possible to blur the foreground and the background in contrast to the main subject. In the pupil dividing-type stereoscopic imaging device, since photographing is carried out with a narrowed diaphragm aperture in order to obtain a clear plane image, creative image formation is limited.